Revisiting Batch Norm Initialization

Batch normalization (BN) is comprised of a component followed by an affine transformation and has become essential for training deep neural networks. Standard initialization each BN in network sets the scale shift to 1 0, respectively. However, after we have observed that these parameters do not alter much from their initialization. Furthermore, noticed process can still yield overly large values, which undesirable training. We revisit formulation present new method update approach address aforementioned issues. Experiments are designed emphasize demonstrate positive influence proper on performance, use rigorous statistical significance tests evaluation. The be used with existing implementations at no additional computational cost. Source code available https://github.com/osu-cvl/revisiting-bn-init .